arXiv:cs/0503044vl [cs.AI] 18 Mar 2005 


Generating Hard Satisfiable Formulas by Hiding Solutions Deceptively 


Haixia Jia and Cristopher Moore and Doug Strain 

Computer Science Department 
University of New Mexico 

{hjia,moore}@cs.unm.edu, doug.strain@gmail.com 


Abstract 

To test incomplete search algorithms for constraint satisfac¬ 
tion problems such as 3-SAT, we need a source of hard, but 
satisfiable, benchmark instances. A simple way to do this 
is to choose a random truth assignment A, and then choose 
clauses randomly from among those satisfied by A. How¬ 
ever, this method tends to produce easy problems, since the 
majority o f literals point toward the “hidde n” assignment A. 
Last year, (Achlioptas, Jia, & Moore 2004| proposed a prob¬ 
lem generator that cancels this effect by hiding both A and 
its complement A. While the resulting formulas appear to be 
just as hard for DPLL algorithms as random 3-SAT formulas 
with no hidden assignment, they can be solved by WalkSAT 
in only polynomial time. 

Here we propose a new method to cancel the attraction to A, 
by choosing a clause with t > 0 literals satisfied by A with 
probability proportional to for some q < 1. By varying g, 
we can generate formulas whose variables have no bias, i.e., 
which are equally likely to be true or false; we can even cause 
the formula to “deceptively” point away from A. We present 
theoretical and experimental results suggesting that these for¬ 
mulas are exponentially hard both for DPLL algorithms and 
for incomplete algorithms such as WalkSAT. 


Introduction 

To evaluate search algorithms for constraint satisfaction 
problems, we need good sources of benchmark instances. 
Real-world problems are the best benchmarks by definition, 
but each such problem has structures specific to its appli¬ 
cation domain; in addition, if we wish to gather good data 
on how the running times of our algorithms scale, we need 
entire families of benchmarks with varying size and density. 

One way to fill this need is to generate random instances. 
For instance, for 3-SAT we can generate instances with n 
variables and m clauses by choosing each clause uniformly 
from among the 8(g) possibilities. We can then vary 
these formulas according to their size and their density 
r = min. While such formulas lack much of the struc¬ 
ture of real-world instances, they have been instrumental 
in the development and study of new search meth - 
ods such as simulated annealing JJohnson et a/. 19^ . 
the breakout procedure JMorris 1993> . 
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WalkSAT ( ISelman, Kautz, & Cohen 1996t , and Sur¬ 
vey Propagation dMezard & Zecchina 2002> . 

However, if we wish to test incomplete algorithms 
such as WalkSAT and Survey Propagation (SP), we need 
a source of hard, but satisfiable problems. In contrast, 
above a critical density r « 4.27, the random formulas 
defined above are almost certainly unsatisfiable. Random 
formulas at this threshold appear to be quite hard for com- 
plete solvers (tC hee^man, Kanefsky, & Taylor 1991 

Hogg, Huberman, & Williams 19961 
Mitchell, Selman, & Levesque 1992j; but for precisely 


this reason, it is not feasible to generate large prob¬ 
lems at the threshold and then filter out the unsatisfiable 
ones. While other classes of satisfiable CSPs have been 
proposed, such as the quasigroup completion prob¬ 
lem (| Sha'^ Stergio u, & Walsh 1998j IKautz et al. 20011 
[Achlioptas et al. 2000^ , we would like to have problems 
generators that are “native” to 3-SAT. 

A natural way to generate random satisfiable 3-SAT for¬ 
mulas is to choose a random truth assignment A G {0,1}", 
and then choose m clauses uniformly and independently 
from among the 7 (g) clauses satisfied by A. The problem 
with this is that simply rejecting clauses that conflict with 
A causes an unbalanced distribution of literals; in particular, 
on average a literal will agree with its value in the hidden 
assignment 4/7 of the time. Thus, especially when there are 
many clauses, a simple majority heuristic or local search will 
quickly find A. More sophisticated versions of this “hid¬ 


den assignment” s cheme (Asahiro, Iwama, & Miyano 1996 
IVan Gelder 19931 improve matters~somewhat but still leac 
to biased samples. Thus the question is how to avoid this 
“attraction” to the hidden assignment. 

One approach ( [Achlioptas, Jia, & Moore 2004| is to 
choose clauses uniformly from among tlmse th^t are sat¬ 
isfied by both A and its complement A. This is in¬ 
spired by recent work on random A:-SAT and Not-All-Equal 
SAT I Achlioptas & Moore 2002b| ', in which symmetry with 
respect to complementation reduces th^ariance of the num¬ 
ber of solutions; the idea is that A and A cancel each others’ 
attractions out, making either one hard to find. Indeed, the 
resulting formulas appear to take DPLL solvers exponential 
time and, in general, to be just as hard as random 3-SAT 
formulas with no hidden assignment. On the other hand, 
WalkSAT solves these formulas in polynomial time, since 






























Density of solutions with r=6 


after a few variables are set in a way that agrees with one of 
the hidden assignments, neighboring variables develop cor¬ 
relations consistent with these JBarthel et al. 20021 . 

In this paper, we pursue an alternate approach, inspired 
by (lAchFioptas & Peres 2003 , who reweighted the satisfy¬ 
ing assignments in a natural way. We hide just one assign¬ 
ment, but we bias the distribution of clauses as follows: for 
each clause, we choose a random 3-tuple (or more generally, 
a fc-tuple) of variables, and construct a clause with f > 0 lit¬ 
erals satisfied by A with probability proportional to q* for 
some constant g < 1. (Note that the naive formulas dis¬ 
cussed above amount to the case q = 1.) This penalizes 
the clauses which are “more satisfied” by A, and reduces 
the extent to which variable occurrences are more likely to 
agree with A. As we will see below, by choosing q appro¬ 
priately we can rebalance the distribution of literals, so that 
each variable is as likely to appear positively as often as neg¬ 
atively and no longer points toward its value in A. By reduc¬ 
ing q further, we can even make it more likely that a variable 
occurrence disagrees with A, so that the formula becomes 
“deceptive” and points away from the hidden assignment. 

We call these formulas “g-hidden,” to distin¬ 
guish them from the naive “1-hidden” formulas 
discussed above, the “2-hidden” formulas studied 
in ( Achlioptas, Jii, & Moore 2004^ , and the “0-hidden” 
formulas consisting of random 3-SAT formulas with no 
hidden assignment. Like these other families, our g-hidden 
formulas are readily amenable to all the mathematical 
tools that have been developed for studying random fc-SAT 
formulas, including moment calculations and the method 
of differential equations. Below we calculate the expected 
density of satisfying assignments as a function of their 
distance from A, and analyze the behavior of the Unit 
Clause (UC) algorithm on g-hidden formulas. We then 
present experiments on several complete and incomplete 
solvers. We find that our g-hidden formulas are just as 
hard for DPLL algorithms as 0-hidden formulas, and are 
much harder than naive 1-hidden formulas. In addition, 
we find that local search algorithms like Walks AT find 
our formulas much harder than any of these other families, 
taking exponential as opposed to polynomial time. More¬ 
over, the running time of WalkSAT increases sharply as 
our formulas become more deceptive. 


The expected density of solutions 

For a £ [0,1], let Xa be the number of satisfying truth as¬ 
signments in a random g-hidden k-SAA formula that agree 
on a fraction a of the variables with the hidden assignment 
A; that is, their Hamming distance from A is (1 — a)n. We 
wish to calculate the expectation E[2fQ]. 

By symmetry, we can take A to be the all-true assignment. 
In that case, a clause with f > 0 positive literals is chosen 
with probability 

(l-bg)'= - 1 

Let B he a truth assignment where an of the vari¬ 
ables are true and (1 — a)n are false. Then, analogous 
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Figure 1: The nth root f{a) of the expected number of solu¬ 
tions which agree with the hidden assignment on a fraction 
a of the variables. Here fc = 3. The upper part of the figure 
shows f{a) for q = 1, q = 0.618 and q = 0.5 at r = 6. The 
lower part shows f{a) for q = 0.5 and varying r. Note that 
at r = 5.6, we have f{a) < 1 for all a < 1/2. 


to I Achlioptas, Jia, & Moore 2004 1 , we use linearity of ex¬ 
pectation, independence between clauses, the selection of 
the literals in each clause with replacement, and Stirling’s 
approximation for the factorial to obtain (where ^ sup¬ 
presses terms polynomial in n): 


E[X,] = 


an 
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where 

“ a“(l - a)i-“ ~ (l-bg)'=-l 

Looking at Figure [2 we see that the behavior of / near 
a = 1/2 changes dramatically as we vary q. For q = 1 (i.e.. 




















naive 1-hidden formulas), /'(1/2) is positive, giving local 
search algorithms a “push” towards the hidden assignment. 
On the other hand, if q is the positive root q* of 

then /'(l/2) = 0. Analogous to ( |Achlioptas & Peres 2003^ , 
this is also the value of q at which literals are equally likely 
to agree or disagree with A. Intuitively, then, if q = q* 
we would expect a local search algorithm starting from a 
random assignment—for which a is tightly concentrated 
around 1 /2—to have no local information telling it in which 
direction the hidden assignment lies. We call these q*- 
hidden formulas balanced', for k = i, q* is the golden ratio 
(^5- l)/2 = 0.618... 

For smaller values of q such as g = 0.5 shown in Fig¬ 
ure ^ /'(1/2) becomes negative, and we expect a local 
search algorithm starting at a random assignment to move 
awa'y from A. Indeed, /(a) has a local maximum at some 
a < 1/2, and for small r there are solutions with a < 1/2. 
When r is sufficiently large, however, /(a) < 1 for all 
a < 1/2, and the probability any of these “alternate” so¬ 
lutions exist is exponentially small. We conjecture that for 
each q < q* there is a threshold rdq) at which with high 
probability the only solutions are those close to A. Set¬ 
ting max{/(a) | a < 1/2} = 1 yields an upper bound 
on rc(g), which we show in Figure 0] below. For instance, 
rc(0.5) < 5.6 as shown in Figure[2 

We call such formulas deceptive, since local search al¬ 
gorithms such as Walk SAT, DPLL algorithms such as 
z Chaff that use a majority heuristic in their splitting rule, 
and message-passing algorithms such as SP will presumably 
search in the wrong direction, and take exponential time to 
cross the local minimum in /(a) to find the hidden assign¬ 
ment. Our experiments below appear to confirm this intu¬ 
ition. In addition, all three types of algorithms appear to en¬ 
counter the most difficulty at roughly the same density Tc (q), 
where we conjecture the “alternate” solutions disappear. 


In this section, we analyze the performance of UC on our 
g-hidden formulas. Specifically, we show that in the bal¬ 
anced case where g = g*, UC fails for r > 8/3 just as for 
0-hidden formulas. Based on this, we conjecture that the 
running time of S, and other simple DPLL algorithms, is 
exponentially large for our formulas at the same density as 
for 0-hidden ones. 

As in l |Achlioptas, Jia, & Moore 2004|| , we analyze the 
behavior of UC on arbitrary initial distributions of 3-SAT 
clauses using the method of differential equations. For sim¬ 
plicity we assume that A is the all-true assignment. A round 
of UC consists of a “free step,” in which we satisfy a random 
literal, and the ensuing chain of unit-clause propagations. 
For 0 < i < 3 and 0 < j < i, let Sij = Sijn be the num¬ 
ber of clauses of length i with j positive literals and i — j 
negative ones, and Si = ^i,j- Let X = xn he the number 

of variables set so far, and let mr and mp he the expected 
number of variables set true and false in a round. Then we 
can model the discrete stochastic process of the Sij with the 
following differential equations for the Sij: 


dS: 


3,j 


dx 

dg2j 

dx 


3^3,J 
1 — a; 


1 — X 


( 1 ) 


2s2,j mF(j + I)s3,j+1 + mT(3 - j)s3j 


(mr + mF)(l — x) 


The unit clauses are governed by a two-type branching pro¬ 
cess, with transition matrix 


M = 


1 


1 — a; 


S2.1 

2S2,2 


2S2,0 

S2.1 


As in I Achlioptas & Moore 2002a!i, as long as the largest 
eigenvalue of M is less than 1, the branching process is sub- 
critical, and summing over the round gives 




Unit Clause heuristic and DPLL algorithms 

Unit Clause (UC) is a linear-time heuristic which perma¬ 
nently sets one variable in each step as follows: if there 
are any unit clauses, satisfy them; otherwise, pick a ran¬ 
dom literal and satisfy it. For random 3-SAT formulas, UC 
succeeds with constant probability for r < 8/3, and fail s 
with high probability for r > 8/3 dChao & Franco 198^ . 
UC can be thought as the first branch of a simple 
DPLL algorithm S, whose splitting rule takes a ran¬ 
dom unset variable and tries its truth values in random 
order; thus UC succeeds if S succeeds without back¬ 
tracking. On the other hand, dCocco & Monasson 20041 
ICocco et al. 200.51 showed that S'’s expected running 
time is exponential in n for any r > 8/3; see 

also (Achlioptas, Beame, & Molloy 2001', who used lower 
bounds on resolution complexity to show that S takes expo¬ 
nential time with high probability if r > 3.81. In general, 
it appears that simple DPLL algorithms begin to take expo¬ 
nential time at exactly the density where the corresponding 
linear-time heuristic fails. 


We then solve the equation o with the initial conditions 
S 3 ,o = 0 and 


- 

bv (1 + 

for 0 < / < 3. In the balanced case g = g*, we find that 
UC succeeds on q-formulas with constant probability if and 
only if r < 8/3, just as for 0-hidden formulas. The rea¬ 
son is that, as for 2-hidden formulas, the expected number 
of positive and negative literals are the same throughout the 
process. This symmetry causes UC to behave just as it would 
on random 3-SAT formulas without a hidden assignment. 

We note that for g < g*, UC succeeds at slightly higher 
densities, at which it can find one of the “alternate” solutions 
with a < 1/2. At higher densities where these alternate so¬ 
lutions disappear, our experimental results below show that 
these “deceptive” formulas take DPLL algorithms exponen¬ 
tial time, and for r > (g) they are harder than 0-hidden 

formulas of the same density. 






















Figure 2; The upper part of the figure shows zChaf f ’s me¬ 
dian running time over 49 trials on 0-hidden, 1-hidden and 
g-hidden formulas with n — 200 and r ranging from 4.0 
to 8.0. The lower part shows the median running time with 
r = 5.5 and n ranging from 50 to 300. 


Experimental results 

DPLL 

In this section we discuss the behavior of DPLL solvers on 
our g-hidden formulas. W e focus on z Cha ff ( |Zhang our 
results from OKsolver dKullmann 2002> are qualitatively 
similar. Figure |2] shows zChaf f’s median running time on 
0-hidden, 1 -hidden, and g-hidden formulas for various val¬ 
ues of g. We see the following phenomena: 

Our g-hidden formulas with q = q* = 0.618... are about 
as hard as 0-hidden ones, and peak in complexity near the 
satisfiability threshold. This is consistent with the picture 
given in the previous two sections: namely, that these “bal¬ 
anced” formulas make it impossible for algorithms to feel 
the attraction of the hidden assignment. In contrast, naive 
1-hidden formulas are far easier, since the attraction to the 
hidden assignment is strong. 

The g-hidden formulas with g < g* are the most interest¬ 


ing ones. The hardness of these formulas shows two phases: 
at low density they are relatively easy, and their hardness 
peaks at a density r'c(g). Above r^q) they take exponen¬ 
tial time; as for 0-hidden formulas, although as r increases 
further the coefficient of the exponential decreases as the 
clauses generate contradictions more quickly. 

We believe that this peak r^q) is the same threshold den¬ 
sity defined earlier (see Figure 0 below) above which the 
only solutions are those close to the hidden assignment. The 
situation seems to be the following: below r^q), there are 
“alternate” solutions with a < 1/2, and zChaf f is led to 
these by its splitting rule. Above rdq), these alternate solu¬ 
tions disappear, and zChaf f takes exponential time to find 
the vicinity of the hidden assignment, since the formula de¬ 
ceptively points in the other direction. Moreover, for a fixed 
r above r^q) these formulas become harder as g decreases 
and they become more deceptive. 

To illustrate this further, the lower part of Figure|2shows 
zChaf f’s median running time on 0-hidden formulas, 1- 
hidden formulas, and g-hidden formulas for g = g* (bal¬ 
anced) and g = 0.3 (deceptive). We also compare with the 
2-hidden formulas of (Achlioptas, Jia, & Moore 2004 . We 
fix r = 5.5, which appears to be above r^q) for both these 
values of g. At this density, the 0-hidden, 2-hidden, and 
balanced g-hidden formulas are all comparable in difficulty, 
while 1-hidden formulas are much easier and the deceptive 
formulas appear to be somewhat harder. 


SP 

Survey Propagation or SP JMezard & Zecchina 20021 is a 
recently introduced incomplete solver based on insights 
from the replica method of statistical physics and a general¬ 
ization of belief propagation. We tested SP on 0-hidden for¬ 
mulas and g-hidden formulas for different values of g, using 
n = 10^ and varying r. For 0-hidden formulas, SP succeeds 
up to r = 4.25, quite close to the satisfiability threshold. 
For g-hidden formulas with g = g*, SP fails at 4.25 just as it 
does for 0-hidden formulas, suggesting that it finds these for¬ 
mulas exactly as hard as 0-hidden ones even though they are 
guaranteed to be satisfiable. For naive 1-hidden formulas, 
SP succeeds at a significantly higher density, up to r = 5.6. 

Presumably the naive 1-hidden formulas are easier for SP 
since the “messages” from clauses to variables, like the ma¬ 
jority heuristic, tend to push the algorithm towards the hid¬ 
den assignment. In the balanced case q = q*, this attrac¬ 
tion is successfully suppressed, causing SP to fail at essen¬ 
tially the same density as for 0-hidden formulas, close to the 
satisfiability threshold, even though our g-hidden formulas 
continue to be satisfiable at all densities. In contrast, the 
2-hidden formulas of ( |Achlioptas, JFa, & Moore 2004j are 
solved by SP up to a somewhat higher density r « 4.8. Thus 
it seems that the reweighting approach of g-hidden formulas 
does a better job of confusing SP than hiding two comple¬ 
mentary assignments does. 

For g < g*, SP succeeds up to somewhat higher densities, 
each of which matches quite closely the value r^q) at which 
zChaf f’s running time peaks (see Figure |4]below). Build¬ 
ing on our conjecture that this is the density above which the 
only solutions are those close to the hidden assignment, we 
























guess that SP succeeds for r < rdq) precisely because the 
local gradient in the density of solutions pushes it towards 
the “alternate” solutions with a < 1/2. Above rdq), these 
solutions no longer exist, and SP fails because the clauses 
send deceptive messages, demanding that variables be set 
opposite to the hidden assignment. 


WalkSAT 


We conclude with a local search algorithm, WalkSAT. For 
each formula, we did up to 10^ restarts, with 10^ steps per 
attempt, where each step does a random or greedy flip with 
equal probability. In the upper part of Figure|3we measure 
WalkSAT’s performance on g-hidden formulas with a range 
of values of q, including q — 1, q = q*, and deceptive val¬ 
ues of q ranging from 0.2 to 0.5. We used n = 200 and 
let r range from 4 to 8. Even for these relatively small for¬ 
mulas, we see that for the three most deceptive values of q, 
there is a density at which the median running time jumps to 
10®, indicating that WalkSAT fails to solve these formulas. 
For instance, g-hidden formulas with q — 0.4 appear to be 
unfeasible for WalkSAT for, say, r > 5. 

We believe that, consistent with the discussion above, lo¬ 
cal search algorithms like WalkSAT greedily follow the 
gradient in the density of solutions f{a). For q < q*, 
this gradient is deceptive, and lures WalkSAT away from 
the hidden assignment. At densities below r^q), there are 
many alternate solutions with a < 1/2 and WalkSAT finds 
one of them very easily; but for densities above r^q), the 
only solutions are those near the hidden assignment, and 
WalkSAT’s greed causes it to wander for an exponentially 
long time in the wrong region. This picture is supported by 
the fact that, as Figure 0] shows below, the density at which 
WalkSAT’s running time jumps upward closely matches the 
thresholds rdq) that we observed for zChaf f and SP. 

The lower part of Figure^looks at WalkSAT’s median 
running time at a fixed density as a function of n. We com¬ 
pare 1-hidden and 2-hidden formulas with g-hidden ones 
with q — q* and two deceptive values, 0.5 and 0.3. We 
choose r = 5.5, which is above r^q) for all three val¬ 
ues of q. The running time of 1-hidden and 2-hidden for¬ 


mulas is only polynomial (Achlioptas, Jia, & Moore 2004 
IBarthel et al. 2QQ2\ . In contrast, even in the balanced case 
q = q*, the running time is exponential, and the slope of 
this exponential increases dramatically as we decrease q and 
make the formulas more deceptive. We note that it might 
be possible to develop a heuristic analysis of WalkSAT’s 
running time in the deceptive case using the methods 
of (|Semerjian & Monasson 2004jlCocco et al. 20051. 


The threshold density 

As we have seen, there appears to be a characteristic density 
rdq) for each value of q < q* at which the running time of 
DPLL algorithms like zChaf f peaks, at which WalkSAT’s 
running time becomes exponential, and at which SP ceases 
to work. We conjecture that in all three cases, the key phe¬ 
nomenon at this density is that the solutions with a < 1/2 
disappear, leaving only those close to the hidden assign¬ 
ment. Figure 13 shows our measured values of rdq), and 



Figure 3; The upper part shows WalkSAT’s median run¬ 
ning time over 49 trials on g-hidden formulas with n = 200 
and r ranging from 4 to 8; the lower part shows the median 
running time with r = 5.5 and n ranging from 50 to 600. 


indeed they are quite close for the three algorithms. We also 
show the analytic upper bound on rdq) resulting from set¬ 
ting max{/(a) | a < 1/2} = 1, above which the expected 
number of solutions with a < 1/2 is exponentially small. 


Conclusions 


We have introduced a simple new way to hide solu¬ 
tions in 3-SAT problems that produces instances that are 
both hard and satisfiable. Unlike the 2-hidden formulas 
of ( [Achlioptas, jTa,~& Moore 2004[ where the attraction of 
the hidden assignment is cancelled by also hiding its com¬ 
plement, here we eliminate this attraction by reweighting the 
distribution of clauses as in (fAchlioptas & Peres 2003 1 . In¬ 
deed, by going beyond the value of the parameter q that 
makes our q-hidden formulas balanced, we can create de¬ 
ceptive formulas that lead algorithms in the wrong direction. 
Experimentally, our formulas are as hard or harder for DPLL 
algorithms as 0-hidden formulas, i.e., random 3-SAT formu- 
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Figure 4: The density rdq) at which the running time of 
zChaf f peaks, WalkSAT peaks or exceeds 10® flips, and 
SP stops working. We conjecture all of these events occur 
because at this density the alternate solutions with a < 1/2 
disappear, leaving only those close to the hidden assignment. 
Shown also is the analytic upper bound described in the text. 


las without a hidden assignment; for local search algorithms 
like WalkSAT, they are much harder than 0-hidden or 2- 
hidden formulas, taking exponential rather than polynomial 
time. Our formulas are also amenable to all the mathemat¬ 
ical tools developed for the study of random 3-SAT; here 
we have calculated their expected density of solutions as a 
function of distance from the hidden assignment, and used 
the method of differential equations to show that UC fails for 
them at the same density as it does for 0-hidden formulas. 
We close with several exciting directions for future work: 

1. Confirm that there is a single threshold density rdq) at 
which a) the alternate solutions far from the hidden as¬ 
signment disappear, b) the running time of DPLL algo¬ 
rithms is maximized, c) SP stops working, and d) the run¬ 
ning time of WalkSAT becomes exponential. 

2. Prove that simple DPLL algorithms take exponential time 
for r > rdq), in expectation or with high probability. 

3. Calculate the variance of the number of solutions as a 
function of a, and giving improved upper and lower 
bounds on the distribution of solutions and rdq)- 
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